Vers la transcription automatique de gestes du soundpainting pour l'analyse de performances interactives

L'analyse objective et la documentation de performances interactives est souvent délicate car extrêmement complexe. Le Soundpainting, langage gestuel dédié à l'improvisation guidée de musiciens, d'acteurs, ou de danseurs, peut constituer un terrain privilégié pour cette analyse. Des gestes prédéfinis sont produits pour indiquer aux improvisateurs le type de matériel souhaité. La transcription des gestes en vue de la documentation de performances semble tout à fait réalisable mais très fastidieuse. Dans cet article, nous présentons un outil de reconnaissance automatique de gestes dédié à l'annotation d'une performance de soundpainting. Un premier prototype a été développé pour reconnaître les gestes filmé par une caméra de type Kinect. La transcription automatique de gestes pourrait ainsi mener à diverses applications, notamment l'analyse de la pratique du soundpainting en général, mais également la compréhension et la modélisation de performances musicales interactives

Soundscape visualization: a new approach based on automatic annotation and Samocharts

International audienceThe visualization of sounds facilitates their identification and classification. However, in the case of audio recording websites, the access to a sound is usually based on the metadata of the sounds, i.e.sources and recording conditions. As sonic environments, or soundscapes, are mostly composed of multiples sources, their compact description is an issue that makes difficult the choice of an item in a sound corpus. The time-component matrix chart, which is abbreviated as TM-chart, has been proposed recently as a tool to describe and compare sonic environments. However their process of creation is based on a subjective annotation that makes their creation time-consuming. In this paper, we present a new method for urban soundscape corpus visualization. In the context of the CIESS project, we propose amochart: an extension of the TM-chart that is based on sound detection algorithms. We describe three original algorithms that allow the detection of alarms, footsteps, and motors. Samocharts can be computed from the results of these algorithms. This process is applied to a concrete case study: 20 urban recordings of 5 minutes each, from different situations (places and time). An application case shows that Samocharts allow an identification of different situations. Finally, the whole method provides a low-cost tool for soundscape visualization that can easily be applied to the management and use of a sound corpus

Caractérisation et reconnaissance de sons d'eau pour le suivi des activités de la vie quotidienne. Une approche fondée sur le signal, l'acoustique et la perception

Avec le vieillissement de la population, le diagnostic et le traitement des démences telle que la maladie d'Alzheimer constituent des enjeux sociaux de grande importance. Le suivi des activités de la vie quotidienne du patient représente un point clé dans le diagnostic des démences. Dans ce contexte, le projet IMMED propose une utilisation innovante de la caméra portée pour le suivi à distance des activités effectuées. Nous avons ainsi travaillé sur la reconnaissance de sons produits par l'eau, qui permet d'inférer sur un certain nombre d'activités d'intérêt pour les médecins, dont les activités liées à l'alimentation, à l'entretien, ou à l'hygiène. Si divers travaux ont déjà été effectués sur la reconnaissance des sons d'eau, ils sont difficilement adaptables aux enregistrements de la vie quotidienne, caractérisés par un recouvrement important de différentes sources sonores. Nous plaçons donc ce travail dans le cadre de l'analyse computationnelle de scènes sonores, qui pose depuis plusieurs années les bases théoriques de la reconnaissance de sources dans un mélange sonore. Nous présentons dans cette thèse un système basé sur un nouveau descripteur audio, appelé couverture spectrale, qui permet de reconnaître les flux d'eau dans des signaux sonores issus d'environnements bruités. Des expériences effectuées sur plus de 7 heures de vidéo valident notre approche et permettent d'intégrer ce système au sein du projet IMMED. Une étape complémentaire de classification permet d'améliorer notablement les résultats. Néanmoins, nos systèmes sont limités par une certaine difficulté à caractériser, et donc à reconnaître, les sons d'eau. Nous avons élargi notre analyse aux études acoustiques qui décrivent l'origine des sons d'eau. Selon ces analyses, les sons d'eau proviennent principalement de la vibration de bulles d'air dans l'eau. Les études théoriques et l'analyse de signaux réels ont permis de mettre au point une nouvelle approche de reconnaissance, fondée sur la détection fréquentielle de bulles d'air en vibration. Ce système permet de détecter des sons de liquide variés, mais se trouve limité par des flux d'eau trop complexes et bruités. Au final, ce nouveau système, basé sur la vibration de bulles d'air, est complémentaire avec le système de reconnaissance de flux d'eau, mais ne peux s'y substituer. Pour comparer ce résultat avec le fonctionnement de l'écoute humaine, nous avons effectué une étude perceptive. Dans une expérience de catégorisation libre, effectuée sur un ensemble important de sons de liquide du quotidien, les participants sont amenés à effectuer des groupes de sons en fonction de leur similarité causale. Les analyses des résultats nous permettent d'identifier des catégories de sons produits par les liquides, qui mettent en évidence l'utilisation de différentes stratégies cognitives dans l'identification les sons d'eau et de liquide. Une expérience finale effectuée sur les catégories obtenues souligne l'aspect nécessaire et suffisant de nos systèmes sur un corpus varié de sons d'eau du quotidien. Nos deux approches semblent donc pertinentes pour caractériser et reconnaître un ensemble important de sons produits par l'eau.The analysis of instrumental activities of daily life is an important tool in the early diagnosis of dementia such as Alzheimer. The IMMED project investigates tele-monitoring technologies to support doctors in the diagnostic and follow-up of the illnesses. The project aims to automatically produce indexes to facilitate the doctor’s navigation throughout the individual video recordings. Water sound recognition is very useful to identify everyday activities (e.g. hygiene, household, cooking, etc.). Classical methods of sound recognition, based on learning techniques, are ineffective in the context of the IMMED corpus, where data are very heterogeneous. Computational auditory scene analysis provides a theoretical framework for audio event detection in everyday life recordings. We review applications of single or multiple audio event detection in real life. We propose a new system of water flow recognition, based on a new feature called spectral cover. Our system obtains good results on more than seven hours of videos, and thus is integrated to the IMMED framework. A second stage improves the system precision using Gammatone Cepstral Coefficients and Support Vector Machines. However, a perceptive study shows the difficulty to characterize water sounds by a unique definition. To detect other water sounds than water flow, we used material provide by acoustics studies. A liquid sound comes mainly from harmonic vibrations resulting from the entrainment of air bubbles. We depicted an original system to recognize water sounds as group of air bubble sounds. This new system is able to detect a wide variety of water sounds, but cannot replace our water flow detection system. Our two systems seem complementary to provide a robust recognition of different water sounds of daily living. A perceptive study aims to compare our two approaches with human perception. A free categorization task has been set up on various excerpts of liquid sounds. The framework of this experiment encourages causal similarity. Results show several classes of liquids sounds, which may reflect the cognitive categories. In a final experiment performed on these categories, most of the sounds are detected by one of our two systems. This result emphasizes the necessary and sufficient aspect of our two approaches, which seem relevant to characterize and identify a large set of sounds produced by the water

Reconnaissance de sons d'eau pour l'indexation en activités de la vie quotidienne

National audienceL’évaluation de troubles dans la réalisation des activités quotidienne est aujourd’hui utilisée dans le diagnostic des démences, mais se heurte à un manque d’outils objectifs. Pour pallier ce manque, le projet IMMED propose la réalisation de vidéo au domicile du patient et l’indexation automatique de ces vidéos en activités. Ces vidéos indexées permettent aux spécialistes de visualiser les patients effectuer des activités dans leur environnement habituel. Dans ce contexte, de nombreuses tâches quotidiennes ont un rapport avec l’eau : se laver les mains, faire la vaisselle, se brosser les dents, etc. Dans cet article, nous présentons deux méthodes de détection de sons d’eau pour la segmentation automatique en activité. La première méthode, basée sur des descripteurs acoustiques, permet la détection du flot d’eau. Pour reconnaître les autres types de sons d’eau, comme les gouttes, nous présentons également une approche originale qui s’appuie sur des modèles acoustiques des sons de liquide

Water sound recognition based on physical models

International audienceThis article describes an audio signal processing algorithm to detect water sounds, built in the context of a larger system aiming to monitor daily activities of elderly people. While previous proposals for water sound recognition relied on classical machine learning and generic audio features to characterize water sounds as a flow texture, we describe here a recognition system based on a physical model of air bubble acoustics. This system is able to recognize a wide variety of water sounds and does not require training. It is validated on a home environmental sound corpus with a classification task, in which all water sounds are correctly detected. In a free detection task on a real life recording, it outperformed the classical systems and obtained 70% of F-measure

Verbal art as heuristic for semantic analyses:How non-prosodic poetic structure in the verbal art of Muylaq’ Aymara (Muylaque, Peru)

International audienceAymara is an Amerindian language spoken mainly in Peru and Bolivia. To date, relatively little is documented about Aymara verbal art. Accordingly, we analyze a traditional song recorded in the Peruvian highlands. We provide a musical and linguistic analysis of the non-prosodic poetic song structure. We detail the octosyllabic, homeoteleutonic (that is, the final words in a line have the same endings) strategies for line formation, the melodic and rhythmic characteristics, and outline the syntactic, morphological, and semantic strategies used in forming semantic couplets. This reveals semantic categories which would not be apparent in a traditional linguistic analysis. Furthermore, the musical analysis confirms previous works on the misperception of a musical anacrusis. We conclude that rigorous, scientific analyses of verbal art require consideration of the construction of meaning through practice and dialogO Aimará é uma língua que é falada principalmente no Peru e na Bolívia. Até o momento, relativamente pouco foi documentado sobre a arte verbal aimará. Nesse sentido, analisamos uma canção tradicional gravada no altiplano peruano. Oferecemos uma análise musical e linguística da estrutura poética não prosódica da canção. Fornecemos detalhes sobre as figuras retóricas / literárias utilizadas para a produção / formação dos versos: a) o verso octossilábico, b) o homeoteleuton (este último é uma figura retórica que consiste na semelhança no final das palavras finais dos versos) e c) as características melódicas e rítmicas. Também delineamos as estratégias sintáticas, morfológicas e semânticas utilizadas/usadas/presentes na formação dos pares semânticos. Isto revela categorias semânticas que não seriam evidentes / não se manifestariam em uma análise linguística tradicional. Além disso, a análise musical confirma as observações de trabalhos anteriores sobre a percepção equivocada de uma anacruse musical. Concluímos que análises rigorosas e científicas da arte verbal requerem considerar a construção de significados por meio da prática e do diálogo

Two-step detection of water sound events for the diagnostic and monitoring of dementia

International audienceA significant aging of world population is foreseen for the next decades. Thus, developing technologies to empower the independency and assist the elderly are becoming of great interest. In this framework, the IMMED project investigates tele-monitoring technologies to support doctors in the diagnostic and follow-up of dementia illnesses such as Alzheimer. Specifically, water sounds are very useful to track and identify abnormal behaviors form everyday activities (e.g. hygiene, household, cooking, etc.). In this work, we propose a double-stage system to detect this type of sound events. In the first stage, the audio stream is segmented with a simple but effective algorithm based on the Spectral Cover feature. The second stage improves the system precision by classifing the segmented streams into water/non-water sound events using Gammatone Cepstral Coefficients and Support Vector Machines. Experimental results reveal the potential of the combined system, yielding a F-measure higher than 80%

Browsing soundscapes

Browsing soundscapes and sound databases generally relies on signal waveform representations, or on more or less informative textual metadata. The TM-chart representation is an efficient alternative designed to preview and compare soundscapes. However, its use is constrained and limited by the need for human annotation. In this paper, we describe a new approach to compute charts from sounds, that we call SamoCharts. SamoCharts are inspired by TM-charts, but can be computed without a human annotation. We present two methods for SamoChart computation. The first one is based on a segmentation of the signal from a set of predefined sound events. The second one is based on the confidence score of the detection algorithms. SamoCharts provide a compact and efficient representation of sounds and soundscapes, which can be used in different kinds of applications. We describe two application cases based on field recording corpora

Sounding out ecoacoustic metrics: avian species richness is predicted by acoustic indices in temperate but not tropical habitats

Affordable, autonomous recording devices facilitate large scale acoustic monitoring and Rapid Acoustic Survey is emerging as a cost-effective approach to ecological monitoring; the success of the approach rests on the de- velopment of computational methods by which biodiversity metrics can be automatically derived from remotely collected audio data. Dozens of indices have been proposed to date, but systematic validation against classical, in situ diversity measures are lacking. This study conducted the most comprehensive comparative evaluation to date of the relationship between avian species diversity and a suite of acoustic indices. Acoustic surveys were carried out across habitat gradients in temperate and tropical biomes. Baseline avian species richness and subjective multi-taxa biophonic density estimates were established through aural counting by expert ornithol- ogists. 26 acoustic indices were calculated and compared to observed variations in species diversity. Five acoustic diversity indices (Bioacoustic Index, Acoustic Diversity Index, Acoustic Evenness Index, Acoustic Entropy, and the Normalised Difference Sound Index) were assessed as well as three simple acoustic descriptors (Root-mean-square, Spectral centroid and Zero-crossing rate). Highly significant correlations, of up to 65%, between acoustic indices and avian species richness were observed across temperate habitats, supporting the use of automated acoustic indices in biodiversity monitoring where a single vocal taxon dominates. Significant, weaker correlations were observed in neotropical habitats which host multiple non-avian vocalizing species. Multivariate classification analyses demonstrated that each habitat has a very distinct soundscape and that AIs track observed differences in habitat-dependent community composition. Multivariate analyses of the relative predictive power of AIs show that compound indices are more powerful predictors of avian species richness than any single index and simple descriptors are significant contributors to avian diversity prediction in multi-taxa tropical environments. Our results support the use of community level acoustic indices as a proxy for species richness and point to the potential for tracking subtler habitat-dependent changes in community composition. Recommendations for the design of compound indices for multi-taxa community composition appraisal are put forward, with consideration for the requirements of next generation, low power remote monitoring networks